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Abstract 

According to the familiar axiom, the eyes are the window to the soul. However, wearing masks to prevent the spread 
of viruses such as COVID-19 involves obscuring a large portion of the face. Do the eyes carry sufficient information 
to allow for the accurate perception of emotions in dynamic expressions obscured by masks? What about the percep- 
tion of the meanings of specific smiles? We addressed these questions in two studies. In the first, participants saw 
dynamic expressions of happiness, disgust, anger, and surprise that were covered by N95, surgical, or cloth masks or 
were uncovered and rated the extent to which the expressions conveyed each of the same four emotions. Across condi- 
tions, participants perceived significantly lower levels of the expressed (target) emotion in masked faces, and this was 
particularly true for expressions composed of more facial action in the lower part of the face. Higher levels of other 
(non-target) emotions were also perceived in masked expressions. In the second study, participants rated the extent 
to which three categories of smiles (reward, affiliation, and dominance) conveyed positive feelings, reassurance, and 
superiority, respectively. Masked smiles communicated less of the target signal than unmasked smiles, but not more 
of other possible signals. The present work extends recent studies of the effects of masked faces on the perception of 
emotion in its novel use of dynamic facial expressions (as opposed to still images) and the investigation of different 
types of smiles. 
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Introduction 


For most people, life during the COVID-19 pandemic 
may have felt flat and lacking in the full range and com- 
plexity of social stimulation. Indeed, people typically 
see each other’s faces signaling states as varied as dis- 
gust, awe, effort, frustration, and attraction, but during 
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the pandemic, we saw fewer faces, and the faces we did 
see were partly covered by masks. The use of masks, of 
course, will continue as policy makers respond to muta- 
tions of the virus such as the Delta variant, and to future 
pandemics. While the health benefits are undeniable, a 
potential adverse effect of the use of face masks is that it 
reduces the amount of information conveyed by the face. 

The overall loss of visual information is significant. 
Human faces are composed of 43 individual muscles 
that possess the unique property of being connected to 
skin and other muscles, but not to bone. The muscles 
can contract in a variety of combinations, producing 
the temporary changes in the superficial geometry of 
the face that we know as facial expressions (Cattaneo 
& Pavesi, 2014; Ekman et al., 1971). Facial expressions 
are nuanced social signals. They efficiently communi- 
cate behavioral intentions, attitudes, as well as emo- 
tional states (Crivelli & Fridlund, 2014; Jack & Schyns, 
2015; Kret et al., 2020), and a perceiver’s ability to 
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accurately extract meaning from facial expressions 
is important for successful social interaction (Hast- 
ings et al., 2008; Marsh & Blair, 2008; Niedenthal & 
Brauer, 2011). 

Is a person’s ability to accurately process the mean- 
ing of facial expression compromised when the per- 
ceived face is covered by a mask? The question must be 
answered empirically because the possibility that people’s 
internal states and momentary intentions can be perceived 
from the eyes alone is a plausible alternative hypothesis 
(Baron-Cohen et al., 2001; Schmidtmann et al., 2020). 
The current work investigated the perception of emotion 
in dynamic human facial expressions that were fully vis- 
ible versus faces covered by the most common face masks 
in use during the COVID-19 pandemic: N95, surgical, 
and cloth.'In Study 1, we explored the effects of mask 
wearing on expressions of some so-called basic emotions 
(Keltner et al., 2003), and in Study 2 we expanded our 
investigation into the effects of mask wearing on more 
nuanced variations of the human smile (Rychlowska et al., 
2017). 

In theory, perceiving emotions in facial expressions 
relies on at least three processes that can be hampered 
by masks (Wood et al., 2016). First, visual information 
from faces can be matched to mental representations of 
expressions that have been encountered and labeled in the 
past. For example, for most people, downturned lips on a 
face have throughout their lives been repeatedly associ- 
ated with the label “sad” or with behavioral indicators 
of sadness, such as crying. Face masks occlude some 
of the visual information used in this matching process. 
Second, emotion recognition can also rely on proprio- 
ceptive feedback from the perceiver’s face; the feelings 
of mimicking the expression on someone else’s face can 
provide essential input into the process of interpreting the 
expression (Wood et al., 2016). Because masks reduce 
visual information conveyed by the face, they can disrupt 
the production of corresponding facial mimicry in a per- 
ceiver. Finally, people rely on contextual information to 
interpret emotion expressions (Hassin et al., 2013). Face 
masks, because they make expressions more ambigu- 
ous, may increase the use of pre-existing stereotypes and 
expectations about emotions that fit the context, which 
may or may not result in an accurate interpretation of the 
expression (Maringer et al., 2011). 


' N95 masks are particulate filtering face masks in the USA, certified 
by the National Institute of Occupational Safety and Health (NIOSH) 
to filter at least 95% of airborne particles (see Fig. 1). 
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For example, when displaying randomly selected 
portions of the face (i.e., the “bubbles” technique), it 
has been shown that the mouth is a primary diagnos- 
tic area for the accurate perception of the six so-called 
basic emotions as well as pain and neutral expressions 
(Blais et al., 2012). However, participants were still able 
to accurately categorize facial expressions from only the 
eyes, but with less frequency (Saumure et al., 2018). Fur- 
ther, categorizations of target emotions are also impaired 
when ‘freezing’ the eyes, and to a greater extent the 
mouth, during dynamically displayed stimuli (Nusseck 
et al., 2008). These findings suggest that blocking signals 
from the lower portion of the face impairs perceptions 
of emotion. However, these methods do not precisely 
match how we perceive expressions on masked faces. A 
plausible alternative interpretation of these findings is 
that the unnatural techniques used to mask portions of 
the face reduce perceiver’s accuracy beyond that due to 
blocking signals from the face. 

Any adverse effect of information occlusion caused by 
mask-wearing may also depend upon the degree to which 
information from a specific emotion expression—its sig- 
nal—is largely conveyed on the lower or upper part of the 
face (Calvo & Nummenmaa, 2008; Smith et al., 2005). 
One method for describing the facial muscle actions that 
comprise emotion expressions is the Facial Action Cod- 
ing System (FACS; Ekman et al., 2002). In this system, 
expressions of happiness (also termed reward smiles in 
the present report) are comprised of the lip corner puller 
(called action unit [AU] 12) and the cheek raiser, which 
causes crow’s feet (AU 6; see Fig. 2 for the action units 
for each of the four emotions examined here). The disgust 
expression is composed of the nose wrinkler (AU 9), the 
lip corner depressor (AU 15), and the lower lip depres- 
sor (AU 16). In both expressions, a large portion of their 
signal is thus located in the lower part of the face. In 
contrast, the anger expression is composed of the actions 
of three muscles on the upper part of the face (AUs 4, 5, 
and 7, brow lowerer, upper eyelid raiser, eyelid tightener, 
respectively) and one muscle on the lower part of the 
face, the lip tightener (AU 23). Surprise is also composed 
of three facial action units in the upper face (AUs 1, 2, 
and 5; inner brow raiser, outer brow raiser, upper eyelid 
raiser) and one action, jaw drop (AU 26), in the lower 
face. The perception of expressions with predominantly 
lower face action units (happiness and disgust) may be 
impacted by face masks to a greater degree than those 
utilizing predominantly upper face musculature (anger 
and surprise). 

The reasoning thus far suggests that masks reduce 
the amount of a target emotion perceived for some, if 
not all, facial expressions: a smile looks less happy, 
and a disgust expression looks less disgusted. However, 
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there is also reason to expect that occluding the face can 
facilitate the perception of other, non-target emotions as 
well. For example, faces covered by a niqab or a black 
rectangle elicit perceptions of smiles and expressions of 
shame as conveying more non-target negative emotions 
compared with fully visible faces (Fischer et al., 2012). 
Further, when mimicry of another person’s smile was 
inhibited, people were more likely to interpret the smile 
in terms of contextual information, leading to less accu- 
racy in perceiving the genuineness of the smile (Mar- 
inger et al., 2011). 

Several recent studies have investigated the effects of 
face masks on the interpretation of facial expressions. 
Faces expressing happiness, sadness, anger, and disgust 
while covered with masks were misclassified as convey- 
ing alternative emotional states more often than their 
fully visible counterparts (Carbon, 2020). This difference 
was not observed for expressions of fear or neutral emo- 
tion. Marini and colleagues (Marini et al., 2021) found 
that opaque face masks reduced the accuracy of emotion 
perception for happiness, fear, and surprised expressions. 
However, when covered by a transparent mask, emotion 
recognition was not significantly different from fully 
visible faces. Children also display a marked decrease 
in their ability to recognize masked expressions of emo- 
tion. Carbon and Serrano (2021) found that children, 
9-10 years old, were less accurate at recognizing expres- 
sions of disgust, fear, happiness, and sadness when the 
expressions were covered by a face mask. However, they 
find a surprising increase in accuracy for anger and neu- 
tral expressions. 

Taken together, these research findings indicate that 
masked expressions communicate less of the intended emo- 
tion especially if the expression involves significant lower 
face action. While such findings are supportive, all prior 
studies employed static photographs as facial expression 
stimuli, while it has been repeatedly demonstrated that much 
of the information from an emotion expression is conveyed 
through its dynamic properties, that is, the timing of the 
contraction of participating muscles (i.e., facial motion; 
Ambadar et al., 2005; Back et al., 2009; Kamachi et al., 
2013). The present two studies therefore employed dynamic 
facial expression videos rather than static images. Each of 
the three studies discussed above utilized a forced-choice 
paradigm. This design feature necessarily limited possible 
responses to the presented expressions. In contrast, the pre- 
sent study utilized a continuous measure of the extent to 
which each face expressed each of four emotions, which 
allowed respondents to report more nuanced perceptions of 
the expressions. 

In Study 1, we investigated the perception of dynamic 
expressions of happiness, disgust, anger, and surprise that 
were fully visible, covered by one of three types of masks 
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(N95, surgical, or cloth), or occluded by a white rectangle 
which was used to control for attitudes towards mask wear- 
ing. In Study 2, we investigated the perception of different 
types of smiles, including reward, affiliation, and dominance 
smiles as described by Martin and colleagues (Martin et al., 
2017, and see below). Research on masks and emotion per- 
ception has yet to investigate the impact of masks on more 
nuanced social signals such as the three types of smiles. 
Additionally, the use of a mask to protect oneself from the 
spread of COVID-19 has become politicized in the United 
States (Pew Research Center, 2020), so we examined the 
possible moderating role of political attitudes on the effects 
of masks on the perception of dynamic facial expressions 
in both studies. The moderating role of gender of the par- 
ticipant-perceiver was also examined since a recent meta- 
analysis confirmed that women are on average more accurate 
than men in recognizing facial expressions (Thompson & 
Voyer, 2014). 


Study 1: Effects of Masks on Perceiving 
Happiness, Disgust, Anger, and Surprise 


Methods 
Participants 


A total of 252 MTurk workers were recruited to par- 
ticipate in a “survey about facial expressions.” Those 
with lower than a 95% HITT rate were excluded. Of 
the total, 162 MTurk workers ranging in age from 20 
to 73 (M = 36.46, SD = 10.85) provided informed con- 
sent, completed the study, passed the attention checks, 
and were included in the analyses. The final sample 
included 89 males and was composed of individuals of 
whom 116 were White, 32 African American, 9 Asian, 
and 5 Native American, Native Alaskan, or Other. 
Participants received $5 in compensation for complet- 
ing the survey, which took them on average 40 min. 
Compensation was based on past research using Mturk 
and balanced thoughtful participation while maximiz- 
ing sample size. Sample size was preregistered and 
is commensurate with previous research (Carbon, 
2020) that utilized a sample of N= 36 for a single 
mask design. 


Materials 
Masks 


We collected royalty-free images of masks through 
an online image search. Three masks were chosen 
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Fig. 1 Dynamic expression stimuli for study 1. The top panel shows 
still images taken from faces dynamically expressing, from left to 
right, happiness, disgust, anger, and surprise. The lower panels show 


to represent those most commonly worn during the 
COVID-19 pandemic including the N95, a blue surgi- 
cal, and a black cloth mask (Fig. 1). The N95 and blue 
surgical mask were collected from CleanPNG.com and 
the black cloth mask was found through a Google Image 
search for “royalty free black cloth mask images.” We 
created the white rectangle using Microsoft 3D Paint 
and included it as a neutral facial occlusion to control 
for attitudes toward mask wearing. 


Emotion Expression Videos 


Expression videos were selected from a large database 
of short, 1 to 3 s, pre-recorded facial expression videos 
(Rychlowska et al., 2017). The final stimulus set included 
videos of 14 actors (3 Black females, 4 White females, 
4 Black males, 3 White males) expressing smiles, anger, 
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images from the same faces covered by a surgical mask, N95 mask, 
and black cloth mask 


disgust, and surprise (42 total videos). Note that not all 
actors expressed every one of the four emotion expressions. 
As discussed above, we used the expressions of anger and 
surprise, because in their prototypic form they involve more 
facial actions in the upper part of the face, and happiness and 
disgust because they involve more facial action in the lower 
part of the face (Fig. 2). While the videos have previously 
been normed, FACS coding of them was not performed for 
the present study. The fully visible conditions of the study 
served as a manipulation check on the level of target emotion 
communicated by each expression for the present samples. 


Dynamic Emotion Expression Stimuli 


Video stimuli were created using Davinci Resolve 16, a video 
editing application. We selected a single point on each actor’s 
face and used point-tracking to track actors’ movements dur- 
ing the expressions. The location of the point tracking varied 
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Happiness Disgust 


Fig.2 FACS coding of happiness, disgust, anger, and surprise 
expressions. Happiness is characterized by AUs 6, the cheek raiser, 
and 12 the lip corner puller. Disgust expressions are predominately 
comprised of AUs 9 the nose wrinkle, 15 the lip corner puller, and 
16 the lower lip depressor. Anger expressions utilize AUs 4 the brow 


due to differences in individual face morphology, and lumi- 
nance and contrast of the video. The three masks and white 
rectangle were overlayed on each expression video, which 
yielded, together with the unmasked videos, 210 unique 
dynamic expression stimuli. The background of the masks 
was removed, and the masks and white rectangle were sized 
to fit the actor’s face. The white rectangle was sized to conceal 
the approximate amount of the actor’s face as the three masks. 
The top, middle of each mask, and the white rectangle were 
placed on the bridge of the nose, so they extended approxi- 
mately 1-inch below the actor’s eyes (Fig. 3). The masks and 
white rectangle overlay were then matched to the movement 
of the actor which ensured that the mask and white rectangle 
occluded the actor’s face over the duration of the video to 
the same degree (Video stimuli can be found at https://osf.io/ 
rbhxd/?view_only=1f01c0205c884f54b7472fb3a617c 15b). 


Survey 
Data were collected via a survey hosted on the Qualtrics 


online survey platform. Participants completed the study 
in a location and on a device of their choosing. Video 





Fig. 3 Mask location for dynamic emotion expression stimuli. The 
green line in the center of the image represents the tracked point of 
the actor’s face and movement throughout the video 
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Anger Surprise 


lowerer, 5 the upper eyelid raiser, 7 the eyelid tightner, and 23 the 
tightner. Surprise expressions are typically comprised of AUs 1, 2, 5, 
and 26 the inner brow raiser, outer brow raiser, upper eyelid raiser, 
and jaw dropper respectively 


stimuli were embedded in the Qualtrics survey. Given the 
variety of possible viewing modalities, the viewing angle 
and distance, central visual location, size of the video, and 
other variables are unknown and a limitation of the present 
research. Video stimuli of facial expressions (happy, dis- 
gust, anger, surprise) for each actor were crossed with rating 
scale (happiness, disgust, anger, surprise) and face condi- 
tion (mask or visible) resulting in 840 dynamic expression 
stimuli and rating scale pairings. The expression-rating 
scale pairings were first divided into four surveys accord- 
ing to mask type (N95, surgical, cloth, white rectangle), 
with all four surveys including visible face videos. There- 
fore, each survey contained 332 pairings which we fur- 
ther divided into two versions containing half of the video 
stimuli resulting in 168 trials per survey. The surveys were 
constructed such that participants did not see a given actor 
expressing the same emotion with and without a mask. The 
two versions were equated for the race (African Ameri- 
can, White) and gender (male, female) of the actors and 
included four attention check questions. Participants rated 
the emotion expressed in each video by responding to the 
prompt, “How much (happiness/disgust/anger/surprise) is 
this person expressing?”, on a 100-point scale (0 = none to 
100 =a great deal). 

Demographic questions were included at the end of 
the survey. Participants reported their age, gender, and 
race, along with their political ideology on a 7-point scale 
(1 =very conservative to, 7 = very liberal), and geographical 
location. Geographical location was assessed on two dimen- 
sions, rural versus urban, and state and city of residence. The 
researchers also included two validity checks at the end of 
the survey, “Did you watch each video before answering the 
question?” and “Did you experience any problems with the 
videos or survey?”’. 


Procedure 


In Study 1, 162 workers on Mechanical Turk (MTurk; Ama- 
zon’s on-demand micro-task platform) viewed 168 videos of 
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14 different actors displaying happiness, disgust, anger, and 
surprise (factor “expression’’). The actors’ faces displayed neu- 
tral emotion at the beginning of the videos and then changed 
to express a particular emotion. In half of the trials, the face 
was fully visible, whereas on the remaining trials, the face 
was masked (factor “face presentation,” see Fig. 1). The face- 
covering was an N95 mask, a blue surgical mask, a black cloth 
mask, or a white rectangle (factor “mask type”). Mask type 
varied between participants such that participants only saw 
one type of mask or the white rectangle throughout the study 
(final N’s were approximately 40 per mask type condition). 

The four emotion expressions were seen an equal number 
of times on visible and masked trials. Expression videos were 
randomly presented four times, and participants rated the extent 
to which the actor was expressing each emotion: once for hap- 
piness, once for disgust, once for anger, and once for surprise. 
Thus, for every emotion expression on both fully visible and 
masked faces, we acquired ratings of the target emotion as well 
as the three non-target emotions (factor “rating scale”). 


Results 


Analyses revealed the predicted two-way interaction between 
face presentation and rating scale, F(1, 158) =277.52, 
p<0.001, Np = 0.637. Specific tests reported in Fig. 4 indi- 
cate that, as expected, for all expressions, the target emotion 
was perceived as less present in masked versus visible faces. 
That is, happy, disgust, angry, and surprised expressions were 


Fig.4 Emotion ratings of Happy Expressions 
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perceived as less happy, disgusted, angry, and surprised, 
respectively, when masked compared with fully visible. 
Participants also perceived more of the non-target emo- 
tions in the masked faces in almost all cases as reported 
in the caption. Specifically, happy expressions, when 
masked, were rated as expressing more anger ¢(161) =6.77, 
p<0.001, d=0.53, and disgust (161) =7.70, p < 0.001, 
d=0.60, but not more surprise, #(161) =0.79, p=0.43. 
When masked, disgust expressions were rated as express- 
ing more happiness, t(161)=6.46, p< 0.001, d=0.51 anger 
t(161)=6.53, p<0.001, d=0.51, and surprise #(161)=2.90, 
p=0.004, d=0.23. Anger expressions were seen as express- 
ing more happiness, 7(161) =5.28, p< 0.001, d=0.41, 
disgust, (161) =2.69, p=0.008, d=0.21, and surprise 
t(161)=4.21, p<0.001, d=0.33, when masked. Analyses 
revealed a slightly different pattern for surprised expres- 
sions. Surprise expressions were rated as expressing less 
surprise when masked (M =77.1, SE=1.19), compared 
to when fully visible (M = 83.2.1, SE=1.09), b=6.07, 
se= 1.04, 11161) =5.84, p <0.001, d2=0.46. However, fully 
visible surprised faces (M = 30.1, SE=1.50) were rated as 
expressing more of the non-target emotions than masked 
faces (M= 24.9, SE=1.81) b=5.25, se=0.73, (1161) =7.21, 
p<0.001, d=0.57. Decomposing the non-target ratings for 
surprised expressions we found that masked surprised faces 
were rated as more angry b=4.16, se=0.89, t(161) =4.70, 
p<0.001, d=0.37 and more disgusted b=5.68, se=0.95, 
t(161)=5.98, p <0.001, d=0.47. However, masked surprise 
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faces were also rated as less happy, b=25.60, se = 1.68, 
t(161)= 15.27, p<0.001, d=1.97. Taken together, our 
results indicate that masked expressions communicate less 
of the target emotion and more of the non-target emotions 
with the exception of surprised and happy faces which were 
rated as expressing more happiness and surprise respec- 
tively when fully visible as compared to when masked. This 
is likely, in part, due to our surprised expression stimuli 
expressing happy as opposed to fearful surprise. 

The perception of facial expressions occluded by masks 
varied by whether the expression prototypically carries more 
of its signal on the upper versus lower part of the face. There 
was a significant two-way interaction between face presen- 
tation (visible, masked) and facial action predominance 
(upper, lower face), F(1, 158) =7.17, p=0.008, np =0.043, 
indicating that perceiving the target emotion in happy and 
disgust expressions (with more signal in the lower part of the 
face) was compromised to a greater degree than perceiving 
the target emotion in angry and surprised expressions (with 
more signal in the upper part of the face). 

Political attitudes did not uniformly moderate any of the 
findings, nor were they moderated by gender of participant 
or the race of the actor (see S1 Appendix, Appendix 1). Fur- 
ther statistical analyses for Study 1 are also described in S1 
Appendix, Appendix 1. 


Study 2: Effects of Masks on Perceiving 
Reward, Affiliation, and Dominance Smiles 


The first study examined prototypical expressions of some of 
the so-called basic emotions on masked versus fully visible 
faces, but many if not most expressions seen “in the wild” 
are more nuanced. Imagine stopping to pick up after your 
dog on a sidewalk of a bustling urban center. A person wear- 
ing a face mask passes you, makes eye contact, and appears 
to express something. What is it? The person seems to be 
smiling, but are they smiling with disdain or gratitude? Are 
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they commiserating because they are also a dog owner? Your 
classification of the smile will determine your response. 

Smile expressions typically involve the activation of the 
muscle that causes the corners of the lips to rise (i.e., AU 12 in 
the FACS), but there is variability in the degree to which other 
actions are present and even the extent to which a participating 
muscle is activated bilaterally. In fact, recent research supports 
the existence of more than one type of smile, and while all 
types are honest social signals that have predictable effects on 
a perceiver’s physiology and behavior (e.g., Martin et al., 2017, 
2021), they do not all communicate happiness. A social-func- 
tional account of the apparent heterogeneity of human smiles 
holds that different forms of smiles are deployed to accom- 
plish distinct social tasks (Martin et al., 2017; Niedenthal et al., 
2010). Reward smiles reinforce desired behavior in the self and 
others (i.e., they shape behavior), affiliation smiles communi- 
cate that the smiler is not a threat and is open to safe interaction 
(i.e., they invite affiliation), and dominance smiles are used to 
convey criticism and social status (i.e., they are used to nego- 
tiate social hierarchies; Martin et al., 2017). Recent studies 
have provided quantitative descriptions of the forms of these 
three social-functional smiles, which are shown in Fig. 5 (e.g., 
Niedenthal & Brauer, 2011). 

The reward smile, which most people view as express- 
ing happy feelings (and was employed in Study 1), is 
symmetrical and usually involves open lips, whereas the 
dominance smile is viewed as expressing far less happi- 
ness, is quite asymmetrical, and sometimes contains facial 
actions seen in contempt, including the nose wrinkler (AU 
9) and the lip raiser (AU 10). English language speakers 
use the terms “smirk” or “smug smile” to describe this 
combination of features. The affiliation smile contains 
unique actions involving the dimpler (AU 14) and the lip 
presser (AU 24), which pull the upper lip across the teeth 
making for an expression that communicates appeasement 
and approachability. Returning to the previous example, a 
passerby could conceivably communicate reassurance or 
disdain as they observe you picking up after your dog in 





Fig.5 Examples of smile stimuli. Reward (left), affiliation (center), and dominance (right; Martin et al., 2021) 


D Springer 


112 


a dense urban area, or they might even reward you for 
fulfilling what many consider a civic duty. The morpho- 
logical distinction among the smiles is largely evident on 
the lower part of the face. So, detecting the meaning of 
a particular smile could be difficult when the person is 
wearing a mask. 

In Study 2, participants saw videos of reward, affiliation, 
and dominance smiles in both fully visible and masked pres- 
entations. Because the major findings from Study 1 did not 
interact with mask type, only the N95 mask was used in this 
study. Participants rated all smiles in terms of the extent to 
which each conveyed positive feelings, reassurance, and supe- 
riority, which we selected to serve as labels for the signals 
communicated by reward, affiliation, and dominance smiles, 
respectively. Note that while we used labels derived from our 
scientific taxonomy of the smile, these labels might not be 
used by laypeople, who, on the other hand, typically have 
shared knowledge about expressions of the so-called basic 
emotions and their labels. 


Methods 
Participants 


A total of 93 MTurk workers were recruited to participate 
in a “survey about facial expressions.” Those with lower 
than a 95% HITT rate were excluded. Of the total, 60 MTurk 
workers ranging in age from 23 to 68 (M=35.25, SD=9.01) 
provided informed consent, completed the study, passed the 
attention checks, and were included in the analyses. The 
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final sample included 36 male participants, 41 were White, 
17 African American, and 2 Asian or Other. Participants 
received $4 in compensation for completing the study that 
took on average 32 min. Compensation was based on past 
research using MTurk and to encourage thoughtful par- 
ticipation while maximizing sample size. Sample size was 
preregistered and is commensurate with previous research 
(Carbon, 2020) that utilized a sample of N=36 with a six 
groups (emotions) design. 


Materials 
Dynamic Expression Stimuli 


Smile expression videos were selected from the same 
database of pre-recorded facial expression videos as 
Study 1 (Martin et al. 2021; Rychlowska et al., 2017). 
The stimulus set included videos of 14 actors (3 black 
females, 4 white females, 4 black males, 3 white males) 
expressing reward, affiliation, and dominance smiles 
(Fig. 6) totaling 34 stimuli. Note that not every actor 
expressed all three smiles. Because there were no impor- 
tant interactions with mask type found in Study 1, only 
the N95 mask was applied to the smile videos, again 
with DaVinci Resolve 16 and following the same steps 
outlined in Study 1. 

Video stimuli of smile expressions were crossed with rat- 
ing scale (positive feeling, reassurance, superiority) and face 
condition (mask or visible) resulting in 204 dynamic expres- 
sion stimuli and rating scale pairings. 





Fig.6 Masked and visible dynamic smile stimuli. The top panel shows images taken from dynamic videos of, from left to right, reward, affilia- 
tion, and dominance smiles. The lower panel show images from the same faces covered by the N95 mask 
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Survey 


Data were collected via a survey hosted again on the 
Qualtrics online survey platform. Expression stimuli 
were divided into two versions of the survey to ensure 
participants did not see an actor expressing the same type 
of smile both fully visible and masked. Participants were 
randomly assigned to be in one of the two versions of 
the survey and stimuli were randomly presented within 
each version. The two versions were equated for the race 
and gender of the actors. Each version of the survey con- 
tained 102 dynamic expression stimuli which participants 
rated on the same 100-point scale, “How much (positive 
feeling/reassurance/superiority) is this person express- 
ing?” Study 2 included three attention check questions 
and participants completed the same demographics ques- 
tionnaire and validity check questions as Study 1. 


Results 


We started by confirming that smiles of reward, affilia- 
tion, and dominance, when fully visible, were perceived 
as communicating the target signals we had labeled as 
positive feelings (reward smiles), reassurance (affiliation 
smiles) and superiority (dominance smiles) see Fig. 7. 
Reward smiles were seen as signaling significantly more 
positive feelings (M =72.5, SE=1.99) than reassur- 
ance and superiority (M = 59.1, SE=2.43), t(59) = 5.25, 
p<0.001, d=0.68.). Dominance smiles also signaled more 
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superiority (M =64.2, SE=2.33) than positive feelings and 
reassurance (M = 58.2, SE = 2.28), t(59)=3.43, p=0.001, 
d=0.44. However, in this sample, affiliation smiles did not 
signal significantly more reassurance than positive feelings 
and superiority, t(59) = 0.35, p=0.73, d=0.05. 

The interaction between face presentation (visible, 
masked), smile type (reward, affiliation, dominance), and 
rating scale (target, non-target), F(2, 118)=4.79, p=0.01, 
np =0.075, was significant. As tests reported in the caption 
(Fig. 4) indicate, we found support for the hypothesis accord- 
ing to which masks impair perception of the target signal 
in all smile types. However, the non-target signals were not 
perceived more in masked versus fully visible faces. 

These findings were not moderated by political attitudes 
or gender of the participant (see S1 Appendix, Appendix 2). 
Further statistical analyses for Study 2 are also described in 
S1 Appendix, Appendix 2). 


Discussion 


If the eyes are the window to the soul, then masking faces 
during a pandemic should have no measurable consequences 
for the ability to perceive emotion in facial expressions accu- 
rately. Despite some evidence that information from the eyes 
is sufficient for the categorization of mental states (Baron- 
Cohen et al., 1997, 2001; Schmidtmann et al., 2020), infor- 
mal testimony, however, suggests that the eyes alone are not 
enough: Individuals in service jobs regularly report that there 
have been frequent miscommunications about desires and 
concerns of customers due to the disruptive effects of masks 
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Fig.7 Social signals of masked and visible smiles. The graphs illus- 
trate levels of social signals communicated in videos of visible and 
masked faces expressing reward, affiliation, and dominance smiles. 
For reward smiles, masks reduced both the target signal (positive 
feeling; p=.003) and non-target signals (p= .046), however, a mar- 
ginally significant (p= .072) between face presentation and rating 
scale interaction indicated that the effect was stronger for the target 
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signal ratings than non-target signal ratings. For affiliation smiles, 
masks impaired perceptions of both the target (reassurance; p=.06) 
and non-target ratings (p =.002), and the interaction was not signifi- 
cant (p=.40). For dominance smiles, masks impaired perception of 
the target signal (superiority; p<.001) but not of non-target ratings 
(p=.67). The interaction was significant (p= .006). tp<.10, *p<.05, 
**p<.01, ***p <.001 
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since the COVID-19 pandemic started (Bhattarai, 2020). The 
present findings lend empirical support to these reports. 

In Study 1, we found that masked happy, disgust, anger, 
and surprise expressions were perceived as conveying signif- 
icantly less happiness, disgust, anger, and surprise, respec- 
tively, than fully visible faces. The same expressions were 
also perceived as containing more of each of the other emo- 
tions. For example, happy expressions were seen as angrier 
and more disgusted which could have ramifications for both 
the perceiver and expressor. Findings for surprise varied 
from this pattern in one way such that masks obscured the 
perception of the amount of positive emotion (rated as hap- 
piness) conveyed by the surprise expression. 

As expected, the effects of masks on the perception of 
emotion were larger for expressions that contain descrip- 
tively more signal from facial muscle contractions on the 
lower compared to upper part of the face. That is, much of 
the communicative signal from happy and disgust expres- 
sions is present in the contraction of the muscles on the part 
of the face specifically covered by protective masks (i.e., the 
lower part of the face) used during a pandemic. The extrac- 
tion of information was somewhat less compromised for the 
expressions of anger and surprise, which involve more signal 
on the upper portion of the face. 

In Study 2, the interpretation of dynamic smiles of 
reward, affiliation, and dominance that had been validated 
in previous work (Martin et al., 2018, 2021) was also com- 
promised when the smiles were covered by face masks. This 
was observed in particular for reward and dominance smiles. 
When covered by masks, reward smiles were perceived as 
signaling less positive feeling and dominance smiles were 
perceived as signaling less superiority. Ratings of affiliation 
smiles as specifically communicating reassurance (the label 
we used for the signaling of non-threat) were only margin- 
ally lower in masked compared to visible conditions. 

Previous research has found that affiliation smiles have 
their intended effects in social interaction (e.g., increasing 
trust) and are mentally represented as possessing a morphol- 
ogy that is distinct from smiles of reward and dominance 
(Rychlowska et al., 2017). Since the time that we selected 
the term “reassuring” to probe the perception of affiliation 
smiles for use in this study, other research in our labora- 
tory found that the lay term that is most likely to be used 
for this smile is not “reassuring” but “fake” (Martin et al., 
2021). This may be due to the fact that, except for “smirks” 
or “smug smiles” (i.e., dominance smiles in our terminol- 
ogy) which by definition do not signal that the smiler is feel- 
ing happiness, there is strong consensus in folk theory that 
“true” smiles always convey happiness. Thus, even though 
affiliation smiles may serve the function of communicating 
non-threat, they may be called “fake” colloquially because 
they are not, nor are they intended to be, signals of happi- 
ness. In future research, the impact of masked emotional 
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expressions on subsequent social behavior and physiological 
responses would provide a stronger test of the hypothesis 
that the perception of emotion in facial expression is com- 
promised by face masks (cf. Martin et al., 2018). 

In the current study, masked stimuli were created by 
superimposing images of popular masks on existing expres- 
sion stimuli. A limitation of this method is the reduction of 
some facial movement that may be seen when the expressor is 
wearing the mask. The face mask is fixed in size and position 
relative to the face which may limit the perception of lower 
face movement. In light of this limitation, the effect sizes 
discussed here, especially for emotions that predominantly 
utilize the lower part of the face, may overestimate those seen 
in natural settings. Further, the superimposition of masks on 
pre-existing dynamic facial expressions does impact the con- 
tinuity of the stimuli. That is, despite the stimuli approxi- 
mating naturally occurring masked expressions, they may 
be perceived as unnatural. Superimposing masks may draw 
attention to the mask over and beyond the amount seen dur- 
ing social interactions during the COVID-19 pandemic. 


Future Directions 


During the COVID-19 pandemic, people in all countries of 
the world have received recommendations for positive health 
behaviors, and the wearing of face masks is one such recom- 
mendation. This begs the question of whether the perception 
of emotion expressions and facial signals of other internal 
states is hindered by mask wearing across cultures. Recent 
research on culture and emotion suggests that any detrimental 
effects of mask wearing are likely not uniform across cultures. 

Specifically, there are differences in how much empha- 
sis cultures place on the explicit facial display of internal 
states of emotion (Matsumoto et al., 2008). Rychlowska 
and colleagues (Rychlowska et al., 2015) reported that peo- 
ple in countries of the world with ancestrally diverse popu- 
lations (i.e., those in which historical migration patterns 
have produced a society composed of people from many 
different countries) tend to encourage the overt display of 
emotion on the face when emotion is felt internally. In con- 
trast, people in less ancestrally diverse countries tend to 
encourage the suppression of explicit displays of emotion. 
The difference may in part be due to the fact that in places 
of high ancestral diversity there is relatively low consensus 
about which emotions are experienced in which contexts 
(Gelfand et al., 2011; Masuda et al., 2008). Facial expres- 
sions may thus provide information that context does not 
(Niedenthal et al., 2019). This reasoning suggests that there 
will be cultural differences in the effects of mask wearing 
such that masks are more detrimental to the processing 
of facial expression in societies of high compared to low 
ancestral diversity. Further, there is some evidence that 


Affective Science (2022) 3:105-117 


there are cultural differences in gaze patterns during facial 
expression recognition. Specifically, the importance of the 
mouth in the recognition of facial expressions may not be 
universal. Jack et al. (2012) showed that internal represen- 
tations of emotion focus more on the eyes for people from 
East Asian cultures compared to the eyes and the mouth 
for people from Western cultures. Assessment of such dif- 
ferences will necessitate the study of facial expression in 
context rather than in isolation as in the present studies. 

Even within cultures there are likely to be changes in 
any effects of mask wearing over the period of a pandemic. 
Characteristics of both perceivers and mask wearers could 
plausibly contribute to these changes. Over time, perceiv- 
ers may come to make more eye contact with mask wear- 
ers. Eye contact can trigger a more accurate simulation of 
the perceived expression and its corresponding state in the 
perceiver for use in interpreting the emotion signal (Wood 
et al., 2016). Mask wearers may begin to intentionally and 
ultimately automatically increase signals to their under- 
lying emotions in the uncovered (upper) versus covered 
(lower) parts of their faces. Mask wearers may also begin 
to increase the expression of their emotions in modalities 
that are not occluded by masks such as the voice (Laukka, 
2005). In addition, they may begin to use conventional 
gestures to complete or augment emotion signals from 
other expressive modalities (Wood et al., 2019). Given the 
decreased perception of happiness in a mask-covered smile 
observed in the present research, increasing other signals 
of positive emotion will be essential for everyday life. 


Conclusion 


The findings reported here, in combination with other 
recent research (Carbon & Serrano, 2021; Carbon, 2020; 
Marini et al., 2021) support the recommendation that 
people increase the communication of emotion through 
modalities that are not compromised by face masks such 
as verbal methods of communicating emotion and the use 
of conventional gesture. This recommendation may prove 
to be especially useful for emotions that utilize the lower 
(happiness, disgust) as opposed to upper part of the face 
(surprise, anger). Our finding, that non-target emotions 
were rated higher for masked faces, further supports this 
recommendation in cases where other cues may be limited. 
This includes ambiguous social situations which may sup- 
port multiple interpretations of a masked expression. 
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